Online learning from nite training sets

We analyse online gradient descent learning from nite training sets at non-innnitesimal learning rates for both linear and non-linear networks. In the linear case, exact results are obtained for the time-dependent generalization error of networks with a large number of weights N, trained on p = N examples. This allows us to study in detail the eeects of nite training set size on, for example, the optimal choice of learning rate. We also compare online and oine learning , for respective optimal settings of at given nal learning time. Online learning turns out to be much more robust to input bias and actually outperforms oine learning when such bias is present; for un-biased inputs, online and oine learning perform almost equally well. Our analysis of online learning for non-linear networks (namely, soft-committee machines), advances the theory to more realistic learning scenarios. Dynamical equations are derived for an appropriate set of order parameters; these are exact in the limiting case of either linear networks or innnite training sets. Preliminary comparisons with simulations suggest that the theory captures some eects of nite training sets, but may not yet account correctly for the presence of local minima .